Repository navigation
Conversation
| return True | ||
| if value == "false": | ||
| return False | ||
| raise ValueError( |
There was a problem hiding this comment.
This new ValueError is reachable from a code path where nothing catches it, so an unexpected boolean wire form will kill the whole session rather than failing one action.
parameters_from_api_response is called in two places:
JobDetails.from_boto(job_details.py:338), which is preceded by_validate_job_parameters— so a bad bool is rejected by the validator first, and that error is wrapped into aSessionActionErrorsubclass by the caller. Fine.SessionActionQueue.dequeueforTASK_RUNparameters (scheduler/session_queue.py:535). Task parameters are never passed through_validate_job_parameters, and that call sits outside thetry/except (ValueError, RuntimeError)block above it (session_queue.py:516-533) which convertsstep_detailsfailures intoStepDetailsError.
So a bare ValueError propagates out of dequeue(). Session._start_action (sessions/session.py:654-658) only catches SessionActionError, so it escapes to Session._run → Session.run, which sets _stop_fail_message, sets the stop event and re-raises — tearing down the entire session and cancelling all remaining queued actions, instead of reporting a single FAILED action with a useful message.
Before this change, value["bool"] for a task parameter was passed through verbatim and could not raise, so this is a new failure mode — and it is precisely the one the rollout makes likely: if the service ever emits a bool form other than exactly true/false (or native bool) for a task parameter, workers hard-fail sessions.
Suggest moving the parameters_from_api_response call at session_queue.py:535 inside a try/except (ValueError, RuntimeError) that raises StepDetailsError(action_id, SessionActionLogKind.TASK_RUN, str(e), step_id=step_id, task_id=task_id), matching how step_details resolution failures are already handled a few lines above. That keeps the strict validation but degrades to a graceful per-action failure.
| param_value = ParameterValue(type=ParameterValueType.LIST_BOOL, value=value["boolList"]) | ||
| param_value = ParameterValue( | ||
| type=ParameterValueType.LIST_BOOL, | ||
| value=[_bool_from_api_response(item) for item in value["boolList"]], |
There was a problem hiding this comment.
Separate from the ValueError reachability point above: this list comprehension now iterates value["boolList"], where previously the value was passed through verbatim without being touched.
That introduces a TypeError (not ValueError) for a non-iterable boolList — e.g. {"boolList": None} raises TypeError: NoneType object is not iterable. For task-run parameters (scheduler/session_queue.py:535) there is no _validate_job_parameters pass and nothing catches it, so it escapes dequeue() past Session._start_actions except SessionActionError and tears down the whole session.
Worth calling out because a try/except (ValueError, RuntimeError) wrapper — the pattern used for the other job-entity errors in dequeue — would not catch this. If you wrap the call, include TypeError, or shape-check the value before iterating.
The service is migrating boolean task and job parameters from native JSON booleans to string-typed booleans, aligning them with int and float parameters, which are already string-typed on the wire. Worker agents must accept the new form before the service begins sending it. BoolParameter.bool and BoolListParameter.boolList now accept str | bool, and a new _bool_from_api_response helper coerces the value back to a native Python bool at the single wire-decode point in parameters_from_api_response. Open Job Description's expression evaluation requires native bools for these parameter types, so coercing once at the decode boundary keeps the Python and Rust session runtimes consistent without changing either adapter. JobDetails.validate_entity_data's shape checks needed the same treatment, as they run ahead of decode and would otherwise reject the string form before the coercion was reached. String values are matched case-insensitively against the Open Job Description boolean vocabulary: "true", "yes", "on", "1", and "1.0" are True; "false", "no", "off", "0", and "0.0" are False. Anything else raises ValueError. Both forms remain valid, so native booleans keep working indefinitely and jobs created before the migration are unaffected. Signed-off-by: Sean Tang <171081544+seant-aws@users.noreply.github.com>
15729de to
7af2fab
Compare
The boolean coercion added alongside the string-typed wire format raises ValueError, and for task-run parameters that exception had no handler. Task parameters are decoded in SessionActionQueue.dequeue outside the try/except that converts step_details failures, and Session._start_action catches only SessionActionError, so a bare ValueError reached Session.run, which sets the stop event and re-raises -- tearing down the whole session and cancelling every remaining queued action instead of failing one. Wrap the decode call so a failure raises StepDetailsError, matching how step_details resolution failures are already handled in the same method. The boolList branch also iterates its value, which raises TypeError rather than ValueError for a non-list such as None, and a handler catching ValueError would miss it. Shape-check the value before iterating so it raises the same ValueError vocabulary as the rest of the decode path. Validation is unchanged: an invalid boolean still fails its action. Signed-off-by: Sean Tang <171081544+seant-aws@users.noreply.github.com>
| # that would tear down the whole session. | ||
| try: | ||
| task_parameters = parameters_from_api_response(task_parameters_data) | ||
| except (ValueError, RuntimeError) as e: |
There was a problem hiding this comment.
The new try closes the ValueError hole but leaves the adjacent TypeError hole open, so the session-teardown failure mode this block exists to prevent is still reachable.
parameters_from_api_response dispatches on "string" in value, "bool" in value, etc. (job_details.py:111-171) without first checking that value is a dict. Task parameters are never run through _validate_job_parameters — task_parameters_data comes straight from the unvalidated AssignedSession payload at line 534 — so if the service ever sends a parameter whose value is not a container (e.g. {"p": null} or {"p": 5}), the membership test raises TypeError: argument of type NoneType is not iterable, not ValueError. That escapes this except (ValueError, RuntimeError), propagates out of dequeue(), and since Session._start_action (sessions/session.py:653-658) only catches SessionActionError, it tears down the whole session and cancels every remaining queued action.
That is precisely the concern the PR itself raises one file over: the boolList shape-check comment at job_details.py:155-157 says it exists so callers "catching only ValueError" do not miss a TypeError. But only that one boolList instance was patched, while the dispatch chain above it has the same exposure for every parameter type.
Two cheap ways to close the class rather than the single instance:
- add
TypeErrorhere:except (ValueError, RuntimeError, TypeError) as e:, or - shape-check at the top of the
parameters_from_api_responseloop, e.g.if not isinstance(value, dict): raise ValueError(f"Parameter {name} -- expected a dict but got {value!r}"), which also yields a message naming the offending parameter.
The second seems preferable since it fixes both call sites and keeps ValueError as the single decode-failure contract that the docstring and the new boolList check already assume.
| # sets; keeping them as explicit sets (rather than a regex) makes the accepted | ||
| # tokens greppable and self-documenting. | ||
| _TRUE_STRINGS = frozenset({"true", "yes", "on", "1", "1.0"}) | ||
| _FALSE_STRINGS = frozenset({"false", "no", "off", "0", "0.0"}) |
There was a problem hiding this comment.
This hand-enumerated vocabulary is now a hard gate: any spelling the service emits that is not in these two sets fails the action (StepDetailsError for TASK_RUN, ValueError out of validate_entity_data for job details). Before this change the value was passed through verbatim and could not fail. So the sets have to match what the producer actually emits, exactly — and the contents suggest they may not.
The set mixes two different rules. true/false/yes/no/on/off is a token list, but 1, 1.0, 0, 0.0 look like the tail of a numeric rule. If the spec rule is numeric, then 1.00, 01, +1, 0.0000 are all equally valid spellings of the same booleans, and the tests here deliberately assert those are rejected (bool-1.00-rejected, bool-01-rejected, bool-00-rejected). If the rule is a token list, it is not obvious why 1.0/0.0 are members at all. One of the two readings is wrong, and if it is the strict one, workers will fail live tasks on parameter values the service considers valid.
Two things worth doing:
-
Pin the accepted set against the actual producer rather than a prose reading of the spec — ideally
openjd-models own boolean coercion if it exposes one (the repo already pinsopenjd-model >= 0.11.4, < 0.12, and this same decode feedsopenjd.expr). Reusing it removes the divergence risk entirely and means the worker cannot drift when the spec adds a token. Re-deriving the vocabulary in the worker means every future spec addition is a worker-side hard-fail until a new agent ships — and agents in the field are not upgraded on the service is schedule. -
If the set must stay local, consider whether an unrecognized token should really be terminal. Given rollout timing, a permissive path (log a warning and fall back to the pre-change behavior for unknown spellings) fails softer than rejecting, since the downside of a wrong coercion is one mis-evaluated expression while the downside of rejection is a failed customer task.
Also worth noting: _TRUE_STRINGS / _FALSE_STRINGS overlap with two existing vocabularies in this same package — telemetry.py:30 (_TRUE_VALUES = {"true", "yes", "on", "1"}) and the config_file.py settings docs (0, off, f, false, n, no, 1, on, t, true, y, yes). Three different accepted sets for the same concept in one codebase is a drift hazard even if each is individually correct for its own input.
| value = cast(BoolParameter, value) | ||
| param_value = ParameterValue(type=ParameterValueType.BOOL, value=value["bool"]) | ||
| param_value = ParameterValue( | ||
| type=ParameterValueType.BOOL, value=_bool_from_api_response(value["bool"]) |
There was a problem hiding this comment.
High-level question on the approach: after this change bool/boolList become the only parameter types in this function that are not passed through verbatim, and the coercion is what forces the new hard-fail behaviour. It is worth confirming the coercion is actually required before accepting that risk.
Look at what the surrounding branches do with the other scalar types whose wire form is a string:
int->value["int"]passed through as a string (validator:isinstance(v, str))float-> passed through as a stringchunkInt-> passed through as a stringrangeExpr-> passed through as a stringintList/floatList-> passed through aslist[str](validator:_is_str_list)
So the consumer downstream of ParameterValue already accepts un-parsed string forms for every other numeric type, including inside lists. If the service flip is normalizing bool to a string and boolList to list[str], that makes them consistent with int/intList rather than exceptional — which suggests the consumer may well accept them as-is too, and the docstring premise ("Open Job Description expression evaluation requires a native bool") may not hold, or may hold only for a path that native bool was already satisfying by accident before the flip.
This matters because the coercion is not free. It is what introduces the vocabulary gate, and therefore the new "unrecognized spelling fails the action" behaviour that did not exist before. If the string form is in fact accepted downstream, the lower-risk change is: widen the type annotations (which this PR already does), leave the value alone, and widen the validator to accept both forms — no coercion, no new failure mode, no worker-side copy of the spec vocabulary to keep in sync.
Two concrete things that would settle it:
- Point at where a native
boolis required —openjd.modelsParameterValue.valuetype, or the expression evaluator that consumes it. IfParameterValue.valueis annotatedstr, then note that the pre-change code was passing a nativeboolinto astr-typed field, and the coercion is preserving a type error rather than fixing one. - Check the
_v1/Rust path specifically:_to_rust_parameter_values(sessions/runtime/rust.py:122-152) handsvalue.valuestraight to the pyo3TaskParameterValue. If that binding accepts the string form forBOOLthe way it does forINT, the coercion buys nothing on the path that is presumably the migration target.
If the native bool genuinely is required, this all stands as-is and the only open item is the vocabulary itself (separate comment). If it is not, dropping the coercion removes the entire class of rollout risk.
Add differential integration coverage that drives the real wire-decode path through real Open Job Description sessions on both the Python and Rust session runtimes, asserting every accepted boolean wire form resolves to the same canonical text and that the two runtimes agree. Declaring a BOOL or LIST[BOOL] parameter requires the EXPR extension in both the template and the decoder's supported extensions; without it the decoder rejects the parameter outright. This coverage also corrected the coercion docstring. Both runtimes render an uncoerced string to canonical text exactly as they render a native bool, so expression substitution does not require the native type. The coercion stays because it conforms to the documented model contract and validates the wire value at the boundary, which is what the docstring now says. Signed-off-by: Sean Tang <171081544+seant-aws@users.noreply.github.com>
What was the problem/requirement? (What/Why)
The service is changing boolean task and job parameters from native JSON
booleans to string-typed booleans, aligning them with int and float parameters,
which are already string-typed on the wire. Accepted values follow the Open Job
Description specification's case-insensitive boolean vocabulary.
Worker agents must accept the new form before the service begins sending it.
Agents in customer-managed fleets upgrade on their own schedule, so this needs
to ship well ahead of the service-side change.
What was the solution? (How)
BoolParameter.boolandBoolListParameter.boolListacceptstr | bool, and anew
_bool_from_api_responsehelper coerces the value to a native Python boolat the single wire-decode point in
parameters_from_api_response. Coercingthere produces the native type Open Job Description's model documents
expression-extension parameters to carry, normalizes both wire forms to one
representation, and validates the value so an out-of-vocabulary string fails
fast at the boundary.
JobDetails.validate_entity_data's shape checks neededthe same widening, since they run ahead of decode and would otherwise reject the
string form before the coercion was reached.
Strings are matched case-insensitively:
true,yes,on,1, and1.0areTrue;
false,no,off,0, and0.0are False. Anything else raisesValueError.The second commit fixes the blast radius of that
ValueError. Task-runparameters are decoded in
SessionActionQueue.dequeueoutside thetry/exceptthat converts
step_detailsfailures, andSession._start_actioncatches onlySessionActionError, so a malformed boolean would have torn down the entiresession and cancelled every remaining queued action rather than failing one. The
decode call now raises
StepDetailsError, matching the existing handler in thesame method. The
boolListbranch also shape-checks before iterating, so anon-list value raises
ValueErrorrather than aTypeErrorthat handler wouldmiss.
What is the impact of this change?
None until the service starts sending strings — both wire forms are accepted, so
this is inert on its own. Native booleans keep working indefinitely rather than
only during a transition window: jobs created before the change retain persisted
native booleans and have them returned verbatim.
A malformed boolean now fails a single action with a useful message instead of
ending the session.
How was this change tested?
Unit tests cover the full vocabulary and its casing for both scalar and list
forms, native-boolean passthrough, mixed-vocabulary lists, and the empty list.
Rejection cases cover
maybe,"",2,01,1.00, surrounding whitespace,native ints
0and1,None, and a bad list element. Coercion tests asserttype(x) is boolalongside the value. Adequeuetest asserts a malformedboolean surfaces as
StepDetailsErrorwith the step and task ids populated,which fails if the call-site wrapper is removed.
The third commit adds differential integration coverage that drives the real
wire-decode path through real Open Job Description sessions on both the
Python and Rust session runtimes, asserting every accepted wire form resolves to
the same canonical text and that the two runtimes agree — a silent divergence
between runtimes fails the test. Worth noting for reviewers: declaring a
BOOLor
LIST[BOOL]parameter requires theEXPRextension in both the template andthe decoder's supported extensions, otherwise the decoder rejects it outright.
That coverage also corrected the coercion's documented rationale. Both runtimes
render an uncoerced string to canonical text exactly as they render a native
bool, so expression substitution does not require the native type. Whether a
deeper boolean operation would is untested — this version's expression grammar
exposes no boolean operators to exercise. The coercion is retained for contract
conformance and wire validation, which is what the docstring now claims.
Unit suite: 3233 passed, 39 skipped. Integration suite: 102 passed, 4 skipped
(pre-existing unrelated skips).
hatch run lintclean (ruff check,ruff format --check,mypy). The Linux E2E suite is being run against the liveservice on both session runtimes to confirm no regression on the native-boolean
path; the new string form cannot be exercised end-to-end until the service-side
change ships.
Was this change documented?
No customer-facing documentation change. The accepted vocabulary and the reason
coercion is needed are documented in the helper's docstring, and the transitional
unions in
api_models.pycarry comments explaining why both wire forms areaccepted.
Is this a breaking change?
No. Native booleans remain valid input, so no existing caller or in-flight job
is affected.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.